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Preface 



This paper attempts to provide evaluators, administrators, and pol- 
icy makers with the advantages of and methodology of merging formative 
and summative data to enhance summative evaluations. It draws on RMC's 
1980-81 California Statewide Evaluation of Migrant Education. The con- 
cern that evaluations typically fail to obtain the "real story" was the 
underlying challenge for the approach used. The aggregate of 22 separ- 
ate formative field studies was merged with the typical summative data 
base. The formative studies were compelling qualitative efforts in and 
of themselves; hence, they provided excellent sources for fleshing out 
the skeletal results from the summative effort. As a result, practi- 
tioners at all levels have accepted and are using the evaluation find- 
ings. 
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Supplementing Summative Findings with Formative Data 



Introduction 

The educational evaluator, or researcher, customarily chooses among 
qualitative, quantitative, formative, and summative contexts within 
which to design her or his methodology. Such things as selecting in- 
struments, sampling units of analysis, and choosing analysis strategies 
depend heavily upon which combination of these contexts controls the 
orientation of the study. 

A major concern which stimulated this paper is that none of these 
decisions may have anything whatsoever to do with the planning, imple- 
mentation, and outcomes of educational programs, materials, or strate- 
gies. These decisions may not affect education at all. No matter how 
rigorous the method, or how appropriate the context, these choices may 
matter only to evaluators themselves. The educational decision-makers 
tend to be only concerned with the "sound" of the evidence. If the re- 
sults sound reasonable, the decision-maker chooses to use them to his or 
her best advantage. If the results sound unreasonable, the decision- 
maker pokes at method and context until she or he finds something miss- 
ing, or amiss.' The educational decison-makers can always find something 
wrong with the methods or context of an evaluation; but they will only 
do so Vhen the results are surprising or especially contradictory to 
their beliefs or wishes. 

If evaluators depend primarily upon quantiative and summative 
orientations to their study, they run the risk of missing the "real 
story" behind the results. They are left to their own experiences, or 
the experiences of consultants, to provide the interpretation that is so 
badly needed. Or, they leave the study at a descriptive level; and 
escape interpretive errors by leaving interpretation to^the readers of 
the evaluation report, i.e., to the same decision-makers who react 
primarily on the sound of the evidence. This is unfortunate. When 
decision-makers are responsible for interpretation, they tend to inter- 
pret in the direction of their beliefs. 
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However, if evaluators orient their study only in a qualitative and 
formative context, they run the risk of being accused of not relying on 
strong statistically significant techniques. The decision-makers have 
been ritualized by their training to expect quantifiable results based 
on rigorous inferential statistical approaches involving representative 
samples. More likely, they may not fund qualitative or formative evalu- 
ations in the beginning because of costs. Decision-makers may not be 
able to cost justify a study that does not meet their understanding of 
appropriate evaluation methodology. 

But the evaluators are faced with an evermore pressing problem. 
Decision-makers are often too busy to take the time to understand and 
use large or complex evaluation projects; they only want the *'bottom- 
line." This mentality is in direct conflict with the needs of program 
participants, the people "in the trenches" trying to make the educa- 
tional program work. They need information from an evaluation that 
keeps with day-to-day decisions about subcomponents of the program. If 
we plan around their needs, the decision-maker is left with very lit- 
tle. If we plan for the decision-maker, program people are left out. 

To be good, an evalution must address the needs of those associated 
with the prograra, and it must be done in a fair and interpretable man- 
ner. This paper shares some insights about an evaluation project that 
has attempted to merge the various contexts for evaluation methods, both 
qualitative and quantitative, plus both formative and summative. The 
project was not as successful as it could have been; but, it provided 
the participating evaluators and decision-makers ample vision to have 
hope that such projects can and should be done. 

The Evaluation Plan 

* ■ 

The evaluation of migrant education in California is being con- 
ducted by the RMC Research Corporation, under contract with the Califor- 
nia Department of Education. During the study three very important is- 
sues had to be addressed by the evaluation methodology: (1) how to 



obtain the "real story" statewide, (2) how to evaluate a complex multi- 
level supplementary program providing educational and health services to 
an extremely mobile student population, and (3) how to provide the peo- 
ple in the trenches with feedback that helps them improve services to 
students, and at the same time provide administrators and policymakers 
with information that helps with all aspects of running the program. 
The State's Office of Program Evaluation and Research (OPER) knew that 
this was a problem and asked for an evaluation that combined the quali- 
tative richness of local-level formative studies with the quantitative 
rigor of a statewide suramative study. 

The Evaluation Process 

The evaluation of California's migrant education is in its second 
year. What is described here happened during the first year. This 
year's evaluation plan was changed somewhat based on the results of last 
year's experiences. 

The district (13) and regional (9) field-sites were visited at 
three different times. The first visit was primarily a planning visit, 
and accomplished three things: (1) established rapport with the field- 
site personnel, (2) collected relevant planning documents and records, 
and (3) obtained input about the content arfd scope of both levels of the 
evaluation, the" field-site and statewide. A totally unstructured inter- 
view technique was used during the first site visits. 

Input from the first visits was integrated with direction from the 
State, and advice from the study's Advisory Panel. The result was a 
complete listing of the evaluation questions addressing all components 
of the program at all levels . The second and third site visitations 
were comprised primarily of semi-structured interviews centering around 
the evaluation questions, plus systematic reviews of the site's docu- 
ments and records. Unstructured observation accounted for the remaining 
field activity. 



3 

6 



Based on the evaluation questions generated early in the study, 
suinmative questionnaires were developed for the various personnel re- 
sponsible for delivering migrant education services, plus migrant par- 
ents and students, and sent to 230 migrant education districts. Half of 
the districts were asked about 1980-81, and half were asked about 1979- 
80. Except for parents whose return rate was only 34%, the average 
return rate was 65%. Statistical checks for bias between years and 
waves of questionnaires were conducted and rejected at 0.05 level. 

Since the State was against the additional burden of testing, 
certain achievement data were obtained through forms requesting grade 
advancement, graduation, and proficiency exam results from 117 dis- 
tricts. The return rate of these data was 7 5% (this was California' s 
first year of proficiency testing, and districts were asked only to 
pilot-test their instruments). Grade-three achievement data from the 
California Assessment Program (CAP) was also accessed, since migrant 
students had been coded separately. Additional enumerative data were 
collected through access to Migrant Education monthly enrollment 
records, and Migrant Student Record Transfer System (MSRTS) records. 

The evaluation team met following the site visits in a series of 
meetings to discuss, compare, and aggregate findings. Evaluation team 
members responsible for particular nigrant education components led each 
meeting and were responsible for recordkeeping for that meeting. The 
focus of data aggregation surrounded the original evaluation questions 
to be answered, plus whatever new questions emerged during the site 
visits . 

In addition, each member of the evaluation team was responsible for 
writing a final formative report which summarized the findings from 
each site they visited. Each team-member was responsible for four to 
six site reports, depending on their other responsibilities. Each of 
the written reports, as well as the earlier oral reports after each site 
visit (each site was visited three times), were provided directly to the 
person responsible for directing the Program at that site. Additional 
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people were provided these information only upon the direction of this 
person. The strictest of anonymity and confidentiality were followed 
throughout the course of the evaluation. 

In the final analysis, 69 formative oral reports were given during 
the evaluation, and 23 written reports were provided. These figures 
include formative and summative reports to the State. 

Merging the Formative with the Summative 

The unique thing about this evaluation was that it actually used a 
semi-structured case-study approach in each field site, observing, 
interviewing, reviewing documents, and gathering enumerative data, while 
gathering questionnaire, achievement, and other enumerative data from 
samples large enough and representative enough for a Statewide summative 
report covering a specified time period. In other words, the evaluation 
accomplished what it set out to do. Hypotheses or conclusions which 
were derived from the summative data, were embellished, confirmed, or 
countered by the findings that emerged from the field studies. Con- 
versely, hypotheses or tentative conclusions which seemed to emerge from 
the field work, could often be substantiated or rejected through post 
hoc analyses of the summative data. The majority of both sets of data, 
however, was formed through the evaluation questions asked at the begin- 
ning of the study. 

On the good side. This paper will not go into the rich intercourse 
between evaluation team members which continually refined the field-site 
approach, and which ultimately helped each team member with their speci- 
fic site reports. The aim of this paper is to demonstrate where the 
formative studies were and were not helpful with the Statewide summative 
report , and why. 

The first, and perhaps greatest impact of the formative field 
effort, was in the formation of the summative evaluation plan. The sum- 
mative plan was not finalized until after the evaluation team had made 



their first round of site visits. It was at this time that the Advisory 
Panel was convened and offered the major thrus;t of the evaluation ap- 
proach. Many of the probing networks of interview and questionnaire 
items grew out of the discourse following the first round of visits. 
Whereas the larger more general questions could be obtained from the 
State, the subtle intricate uses of detailed probes were derived from 
the feel of what was actually happening or needed by program partici- 
pants in the field. If the real story was what the evaluators were 
after, it continually emerged with increased close contact with the 
field. 

The second impact of the formative was the formulation of a series 
of hypotheses or counterquestions throughout the course of the data col- 
lection phase. As team members talked after each site visit, pieces of 
the puzzle began to fit together, and an understanding of how the pro- 
gram operates, began to grow. 

The third impact of the formative was the growing rapport between 
the evaluators and program participants in the field. A mounting major- 
ity of those contacted, stated increased interest in our findings and 
faith that the findings would be believable. They knew that the re- 
sults, since they were to get their own report on their own project, 
would be useful. However, as time went on, they pressured us to make 

sure the State and other program administrative levels would pay atten- 
tion to the Statewide results. In a sense, the field began to support 
and take ownership in the Statewide summative evaluation as well as 
their own local evaluation. 

The fourth impact came during the analysis of summative data. 
Three things occurred: (1) specific sets of questionnaire data were 
included or excluded because of the now known relevance to what was 
actually occurring in the field; (2) data layouts were often formated or 
centered around questions arising from the field work; and (3) the find- 
ings indicated by the summative data could be traced among the formative 
data for confirmation or clarification. In addition, the Advisory Panel 
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was reconvened to see initial (draft) layouts of the summative data. 
The Panel's questions or concerns prompted discussions which almost in- 
variably required use of formative findings. 

Finally, the fifth impact, and perhaps the most observable, was the 
use of formative data during the writing of the summative report. Sev- 
eral things happened during this stage. The report was initially writ- 
ten in separate components, with each component written by the evalua- 
tion team member responsible for that component. Based on the meetings 
that were held to discuss and aggregate the formative data, each team 
member drafted their components using appropriate portions of the aggre- 
gated formative data. The initial drafts were ther, circulated to the 
other team members, who proofread the sections in terms of their uncf er- 
standing from the 'data, both formative and summative. The basic result 
of this proofreading led to a sort of validation that the formative 
analysis conducted earlier had reached a nearly consensus level, and 
that the report made reasonable references to the formative data. When 
one looks at; the report, one senses that the interpretations of data and 
the subsequent conclusions and recommendations had been grounded in pro- 
gram reality, and not in the "ivory-tower." 

On the bad side . A major problem throughout the study was the fact 
that the formative studies were being conducted by five different eval- 
uators~how could we cope with individual differences in style and inte- 
rests, not to mention biases? To some degree, the staff meetings, re- 
ferred to as "crosstalks," achieved certain levels of conformity, both 
in approach to data collection and in approach to analysis and report- 
ing. The semi-structured interview guides and list of documents to be 
reviewed which arose out of the basic set of evaluation questions, added 
some additional commonality. " Staff training, too, helped to secure more 
consistency across evaluators. And, during the analysis and reporting, 
the evaluators had a chance to delete, question, or add portions of the 
building story. Despite all of these steps, everyone of the evaluators 
felt uneasy and concerned about differences among themselves, such as 
the varying focus of extra probing in the field, the different levels of 



'10 



knowledge in certain areas to be investigated, and diverse styles of 
working with field contacts. Unfortunately, the time and budget for the 
evaluation were too tight to allow teams of evaluators, or to allow the 
project director to monitor field work directly. 

Another problem came from a basic, almost philosophical, difference 
among evaluators regarding how much faith to place in the formative data 
since it came from a small case-study sample of sites. This particular 
problem surfaced again and again, and was a major stumbling block at one 
point during report writing. Some of the evaluators could not place 
enough reliance on the rich data from such a small, perhaps nonrepre- 
sentative, sample. They tended to be more afraid to embellish, confirm, 
clarify, or reject summative results based on formative data. They were 
not afraid, however, to raise hypotheses based on formative data, and 
use summative data to verify or reject. The summative data base, how- 
ever lacked flexibility since it was a fixed set of questions and in- 
formation. Gaps in the summative data collection effort remained gaps 
forever; whereas, the formative data collection allowed the real story 
to emerge, even if the evaluation probes were faulty to begin with. 
This particular dilemma was basically resolved through the circulation 
of component drafts of the report, and through occasional resolutions by 
the project director. 

0 

r. 

The final problem rested with the level of ownership of the evalua- 
tion by Program administrators. As mentioned in the introduction, ad- 
ministrators are often too busy to involve themselves in the details and 
methodology of an evaluation effort. From the evaluators' point-of- 
view, had there been a higher level of ownership , the f ormacive effort 
would have dove- tailed with the summative analysis much better than it 
did. The summative report was hard-hitting and rather assertive, and 
unfortunately it took certain administrators by surprise , which reflects 
badly on the formative. Why? 

The evaluation team was not able, in the final analysis, to achieve 
the degree of rapport at the top level as it had with lower levels of 
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the Program. There was more ownership in the field than at the State. 
And, it could have been a major drawback in subsequent evaluation use. 
Such was not the case. The State-level personnel tended to somehow 
respect the report, and the intent of the evaluation team. The major 
findings of the report were focused upon, and there have been several 
policy or administrative decisions that addressed findings or recommen- 
dations from the report. In fact, the use made of the report has 
spurred the evaluators to realize that if they had improved rapport at 
the top, even more use may have been made of it. They have resolved to 
critically review how they as evaluators can do a better job of involv- 
ing top administrators. 



Summary 

The summative report would have been little more than an enumera- 
tive report without the use of formative data. The formative data rose 
above the usual qualitative data since the formative data collection was 
designed to provide formative information to field participants. The 
fact that field participants could see utility for themselves, caused 
them to take more ownership in providing the "real story." As a result, 
they grew to understand and respect the evaluators sense of the mounting 
story and also grew to take ownership in the Statewide study. This led 
to a rich data base that could be merged with summative data and help 
create a summative report that was based on Program reality. There was 
usefulness in the report. There was use made of it. 
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